Methods and apparatus employing a phase detection autofocus (PDAF) optical system

ABSTRACT

Apparatus and methods employing a PDAF optical system are disclosed herein. An example apparatus includes an image sensor comprising a plurality of pixels. The plurality of pixels include a set of pixels configurable to be imaging pixels or focus pixels. The image sensor is configured to generate image data of a scene based on received light at the plurality of pixels. The example apparatus also includes a processor coupled to the image sensor. The processor may be configured to receive first image data of a first frame of the scene, determine at least one region of interest or region of non-interest of the first frame, select, based on the determined at least one region of interest or region of non-interest, a subset of the set of pixels to be focus pixels, and cause the selected subset of the set of pixels to operate as focus pixels.

CROSS REFERENCE TO RELATED APPLICATION(S)

This application is a Continuation of U.S. Non-provisional applicationSer. No. 16/859,821, entitled “METHODS AND APPARATUS EMPLOYING A PHASEDETECTION AUTOFOCUS (PDAF) OPTICAL SYSTEM” and filed on Apr. 27, 2020,which is expressly incorporated by reference herein in its entirety.

BACKGROUND Technical Field

The present disclosure relates generally to phase detection autofocus(PDAF), and more particularly, to methods and apparatus employing a PDAFoptical system.

INTRODUCTION

Image capture devices, such as digital cameras and mobile devices (e.g.,smartphones, tablets, laptops, etc.) include an imaging system thatincludes an image sensor positioned downstream of one or more opticalcomponents. Typical optical components may include one or more lensesand apertures. The optical components direct light of a scene onto theimage sensor. A processor processes the data captured by the imagesensor to record an image. To record a clear image, the opticalcomponents focus light from the scene onto the image sensor. If thelight is out of focus at the plane of the image sensor, then the imagesensor may capture a blurry image.

Some image capture devices use phase detection autofocus (PDAF) pixelsto perform autofocus. Image capture devices may include an image sensorincluding an array of pixels. The array of pixels may include one ormore imaging pixels and one or more focus pixels (also referred to as“phase detection pixels”) arranged in a pattern. There is currently aneed to improve autofocus technology, including techniques implementedwith a combined image sensor (e.g., an image sensor including imagingpixels and focus pixels).

SUMMARY

The following presents a simplified summary of one or more aspects inorder to provide a basic understanding of such aspects. This summary isnot an extensive overview of all contemplated aspects, and is intendedto neither identify key or critical elements of all aspects nordelineate the scope of any or all aspects. Its sole purpose is topresent some concepts of one or more aspects in a simplified form as aprelude to the more detailed description that is presented later.

In an aspect of the disclosure, a method, a computer-readable medium,and an apparatus are provided. An example apparatus comprises an imagesensor and a processor coupled to the image sensor. The example imagesensor may include a plurality of pixels including a set of pixelsconfigurable to be imaging pixels or focus pixels. The example imagesensor may be configured to generate first image data of a scene basedon received light at the plurality of pixels. The example processor maybe configured to receive the first image data of a first frame of thescene. The example processor may be configured to determine at least oneregion of interest or region of non-interest of the first frame. Theexample processor may be configured to select, based on the determinedat least one region of interest or region of non-interest, a subset ofthe set of configurable pixels to be focus pixels. The example processormay be configured to cause the selected subset of the set ofconfigurable pixels to operate as focus pixels.

In another aspect, disclosed techniques include a method of operationfor autofocus. The example method includes receiving, from an imagesensor, first image data of a first frame of a scene. In some examples,the image sensor may include a plurality of pixels, the plurality ofpixels including a set of pixels configurable to be imaging pixels orfocus pixels. In some examples, the image sensor may be configured togenerate the first image data of the scene based on received light atthe plurality of pixels. The example method also includes determining atleast one region of interest or region of non-interest of the firstframe. Additionally, the example method includes selecting, based on thedetermined at least one region of interest or region of non-interest, asubset of the set of configurable pixels to be focus pixels. Further,the example method includes causing the selected subset of the set ofconfigurable pixels to operate as focus pixels.

In another aspect, disclosed techniques include an apparatus forperforming autofocus. The example apparatus includes means forreceiving, from an image sensor, first image data of a first frame of ascene. In some examples, the image sensor may include a plurality ofpixels, the plurality of pixels including a set of pixels configurableto be imaging pixels or focus pixels. In some examples, the image sensormay be configured to generate the first image data of the scene based onreceived light at the plurality of pixels. The example apparatus alsoincludes means for determining at least one region of interest or regionof non-interest of the first frame. Additionally, the example apparatusincludes means for selecting, based on the determined at least oneregion of interest or region of non-interest, a subset of the set ofconfigurable pixels to be focus pixels. Further, the example apparatusincludes means causing the selected subset of the set of configurablepixels to operate as focus pixels.

To the accomplishment of the foregoing and related ends, the one or moreaspects comprise the features hereinafter fully described andparticularly pointed out in the claims. The following description andthe annexed drawings set forth in detail certain illustrative featuresof the one or more aspects. These features are indicative, however, ofbut a few of the various ways in which the principles of various aspectsmay be employed, and this description is intended to include all suchaspects and their equivalents.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example image sensor including sparse focus pixelsthat may be used to perform PDAF.

FIG. 2 is a block diagram that illustrates an example content generationsystem, in accordance with one or more techniques of this disclosure.

FIG. 3 illustrates a side view of an example configurable pixel of animage sensor, in accordance with one or more techniques of thisdisclosure.

FIGS. 4A and 4B illustrate top view of configurable pixels, inaccordance with one or more techniques of this disclosure.

FIGS. 5A to 5F illustrate top views of focus orientations forconfigurable pixels in which the opacity transitioning material isdivided into four independently-controllable portions, in accordancewith one or more techniques of this disclosure.

FIG. 6 is a block diagram of an image processor, in accordance with oneor more techniques of this disclosure.

FIG. 7 depicts an example image, in accordance with one or moretechniques of this disclosure.

FIG. 8 depicts a region of interest including a plurality of horizontaltextures, in accordance with one or more techniques of this disclosure.

FIG. 9 depicts an example image including an identified region ofinterest, in accordance with one or more techniques of this disclosure.

FIG. 10 illustrates a portion of an image sensor including an array ofpixels, in accordance with one or more techniques of this disclosure.

FIGS. 11 to 15 illustrate example flowcharts of example methods that maybe implemented by the example device of FIG. 2 , in accordance with oneor more techniques of this disclosure.

DETAILED DESCRIPTION

The detailed description set forth below in connection with the appendeddrawings is intended as a description of various configurations and isnot intended to represent the only configurations in which the conceptsdescribed herein may be practiced. The detailed description includesspecific details for the purpose of providing a thorough understandingof various concepts. However, it will be apparent to those skilled inthe art that these concepts may be practiced without these specificdetails. In some instances, well known structures and components areshown in block diagram form in order to avoid obscuring such concepts.

Various aspects of systems, apparatuses, computer program products, andmethods are described more fully hereinafter with reference to theaccompanying drawings. This disclosure may, however, be embodied in manydifferent forms and should not be construed as limited to any specificstructure or function presented throughout this disclosure. Rather,these aspects are provided so that this disclosure will be thorough andcomplete, and will fully convey the scope of this disclosure to thoseskilled in the art. Based on the teachings herein one skilled in the artshould appreciate that the scope of this disclosure is intended to coverany aspect of the systems, apparatuses, computer program products, andmethods disclosed herein, whether implemented independently of, orcombined with, other aspects of the disclosure. For example, anapparatus may be implemented or a method may be practiced using anynumber of the aspects set forth herein. In addition, the scope of thedisclosure is intended to cover such an apparatus or method which ispracticed using other structure, functionality, or structure andfunctionality in addition to or other than the various aspects of thedisclosure set forth herein. Any aspect disclosed herein may be embodiedby one or more elements of a claim.

Although various aspects are described herein, many variations andpermutations of these aspects fall within the scope of this disclosure.Although some potential benefits and advantages of aspects of thisdisclosure are mentioned, the scope of this disclosure is not intendedto be limited to particular benefits, uses, or objectives. Rather,aspects of this disclosure are intended to be broadly applicable todifferent wireless technologies, system configurations, networks, andtransmission protocols, some of which are illustrated by way of examplein the figures and in the following description. The detaileddescription and drawings are merely illustrative of this disclosurerather than limiting, the scope of this disclosure being defined by theappended claims and equivalents thereof.

Several aspects are presented with reference to various apparatus andmethods. These apparatus and methods are described in the followingdetailed description and illustrated in the accompanying drawings byvarious blocks, components, circuits, processes, algorithms, and thelike (collectively referred to as “elements”). These elements may beimplemented using electronic hardware, computer software, or anycombination thereof. Whether such elements are implemented as hardwareor software depends upon the particular application and designconstraints imposed on the overall system.

By way of example, an element, or any portion of an element, or anycombination of elements may be implemented as a “processing system” thatincludes one or more processors (which may also be referred to asprocessing units). Examples of processors include microprocessors,microcontrollers, graphics processing units (GPUs), general purpose GPUs(GPGPUs), central processing units (CPUs), application processors, imagesignal processors (ISPs), digital signal processors (DSPs), reducedinstruction set computing (RISC) processors, systems-on-chip (SOC),baseband processors, application specific integrated circuits (ASICs),field programmable gate arrays (FPGAs), programmable logic devices(PLDs), state machines, gated logic, discrete hardware circuits, andother suitable hardware configured to perform the various functionalitydescribed throughout this disclosure. One or more processors in theprocessing system may execute software. Software can be construedbroadly to mean instructions, instruction sets, code, code segments,program code, programs, subprograms, software components, applications,software applications, software packages, routines, subroutines,objects, executables, threads of execution, procedures, functions, etc.,whether referred to as software, firmware, middleware, microcode,hardware description language, or otherwise. The term application mayrefer to software. As described herein, one or more techniques may referto an application, i.e., software, being configured to perform one ormore functions. In such examples, the application may be stored on amemory, e.g., on-chip memory of a processor, system memory, or any othermemory. Hardware described herein, such as a processor, may beconfigured to execute the application. For example, the application maybe described as including code that, when executed by the hardware,causes the hardware to perform one or more techniques described herein.As an example, the hardware may access the code from a memory andexecute the code accessed from the memory to perform one or moretechniques described herein. In some examples, components are identifiedin this disclosure. In such examples, the components may be hardware,software, or a combination thereof. The components may be separatecomponents or sub-components of a single component.

Accordingly, in one or more examples described herein, the functionsdescribed may be implemented in hardware, software, or any combinationthereof. If implemented in software, the functions may be stored on orencoded as one or more instructions or code on a computer-readablemedium. Computer-readable media includes computer storage media. Storagemedia may be any available media that can be accessed by a computer. Byway of example, and not limitation, such computer-readable media cancomprise a random access memory (RAM), a read-only memory (ROM), anelectrically erasable programmable ROM (EEPROM), optical disk storage,magnetic disk storage, other magnetic storage devices, combinations ofthe aforementioned types of computer-readable media, or any other mediumthat can be used to store computer executable code in the form ofinstructions or data structures that can be accessed by a computer.

Phase detection autofocus (PDAF) techniques are based on measuring theoffset (or phase difference) between two images that were generated withdifferent asymmetric optical apertures. The magnitude and sign of thecalculated phase may be correlated with an amount of defocus, which maybe used to estimate the lens displacement needed for optimal focus.

There are many different types of focus pixel patterns and/orarrangements that may be included in an image sensor. FIG. 1 illustratesan example image sensor 100 including sparse focus pixels that may beused to perform PDAF. An array 102 of pixels 104 may include one or morefocus pixels 106 that are shielded (e.g., using metal shielding) thatmay be used to create left or right images used in PDAF. For example, insome examples using sparse PDAF with metal shielding, one to threepercent of the pixels 104 may be dedicated to left/right PDAF. On eachof the focus pixels 106, a metal shield 108 may be positioned between aphotodiode and a microlens (sometimes referred to as a “u-lens,” an“on-chip lens,” or “OCL”) that creates an effective asymmetric apertureby blocking light at certain interception angles, allowing for thecapture of left or right image information for PDAF purposes. Themicrolens may be used to increase the sensitivity of the underlyingphotodiode towards light. For imaging purposes, the values of the focuspixels 106 may be replaced by an interpolated value of the surroundingpixels, similar to how defective pixels may be corrected.

It should be appreciated that while the above description of the imagesensor 100 includes left and right pairs of focus pixels, in otherexamples, the image sensor 100 may additionally or alternatively includeup and down pairs of focus pixels. For example, the mask or metalshielding of focus pixels may mask top and bottom portions of therespective focus pixels, which may generate up and down (or top andbottom) pairs of images.

In some examples, the focus pixels may include dual photodiode (2PD)pixels where each focus pixel includes two diodes (e.g., a firstphotodiode and a second photodiode adjacent to the first photodiode). Insome examples, the image sensor 100 may include one or more 2PD focuspixels including a left diode and a right diode. In some such examples,the left diode may generate a left image and the right diode maygenerate a right image. In some examples, the image sensor 100 mayinclude all 2PD focus pixels (e.g., without any imaging pixels includinga single photodiode). In some examples, the image sensor 100 may includesparse 2PD focus pixels. For example, the image sensor 100 may include apixel array including a first subset of imaging pixels including asingle photodiode and a second subset of 2PD focus pixels. In some suchexamples, the 2PD focus pixels may be arranged in any manner (e.g.,adjacent to one another, spaced apart from one another, etc.). In someexamples, the focus pixels of the pixel array of the image sensor 100may be left-right pairs of focus pixels or up-down pairs of focuspixels. In some examples, the image sensor 100 may include a combinationof left-right pairs of focus pixels and up-down pairs of focus pixels.

In some examples, the image sensor 100 may include 4PD focus pixelswhere each focus pixel includes four photodiodes (sometimes referred toas “quad-PD” or QPD″ pixels). In some such examples, the 4PD focuspixels may generate two pairs of images (e.g., a pair of left-rightimages and a pair of up-down images).

In some examples, the image sensor 100 may include all 4PD focus pixels(e.g., without any imaging pixels including a single photodiode). Insome examples, the image sensor 100 may include sparse 4PD focus pixelssuch that a subset of the focus pixels of the pixel array of the imagesensor 100 are 4PD focus pixels arranged in any manner (e.g., adjacentto one another, spaced apart from one another, etc.). In some suchexamples, one or more remaining pixels of the pixel array may be imagingpixels and/or 2PD focus pixels.

Once pairs of images have been generated for each focus pixel (e.g., apair of left-right images and/or a pair of up-down images), the imagesmay be compared with one another. For example, phase differences betweensignals generated from the left focus pixels (e.g., the left image) andthe right focus pixels (e.g., the right image) may be compared and anoffset (e.g., a phase disparity) between the signals may be determined.The offset may be used to estimate the lens displacement needed foroptimal focus.

While different image sensors may utilize different patterns and/orarrangements of focus pixels, it should be appreciated that thedistribution of such focus pixels within the image sensor is fixed. As aresult, the density of focus pixels providing focus data (or “phaseinformation”) for PDAF purposes associated with a region of interest inan image is also fixed. However, for some regions of interest (ROIs) inan image (e.g., a face, food, and other ROIs that people consider as“more important”), the density of corresponding focus pixels may be toolow to provide accurate focus data. For other ROIs (e.g., the sky, wall,and other ROIs that people consider of “low importance”), the density offocus pixels within the image sensor that are associated with the ROImay be too high and may lead to a waste of information.

In some examples, the fixed focus orientation (e.g., left-right orup-down) of the focus pixels may result in textures or edges of an imagebeing undetected. For example, focus data provided by focus pixels witha left-right focus orientation may be unable to detect a horizontaltexture within a region of interest. In some examples, if a region ofinterest includes an area of non-interest, such as a saturated area,focus data provided by focus pixels corresponding to the area ofnon-interest may not provide useful information. In some examples, thefixed pattern and/or arrangement of focus pixels may result in artifactswhen de-noising techniques are applied.

In general, example techniques disclosed herein are directed toadjusting focus pixels of an image sensor based on an analysis of ascene. For example, disclosed techniques utilize an image sensor havinga plurality of pixels including a set of pixels configurable to operateas focus pixels or imaging pixels. For example, a configurable pixel mayinclude a photodiode and an opacity transitioning material positionedabove the photodiode. The opacity transitioning material may include afirst opacity transitioning material portion arranged above a firstsection of the photodiode and a second opacity transitioning materialportion arranged above a second section of the photodiode, and where thefirst opacity transitioning material portion and the second opacitytransitioning material portion are independently configurable to beopaque or transparent. In some such examples, the configurable pixel maybe configured to operate as a focus pixel when one of the first opacitytransitioning material portion or the second opacity transitioningmaterial portion is opaque and the other opacity transitioning materialportion is transparent. The configurable pixel may be configured tooperate as an imaging pixel when both opacity transitioning materialportions are transparent.

When the configurable pixel is configured to operate as an imagingpixel, the respective imaging pixel generates imaging data that may beused to capture the scene. When the configurable pixel is configured tooperate as a focus pixel, the respective focus pixel generates focusdata that may be used to perform autofocusing. In some examples, whenthe configurable pixel is configured to operate as a focus pixel, therespective focus pixel may be configured with a focus orientation. Asdescribed above, focus pixels may be arranged in a left-right patternwhere a first focus pixel may operate as a left focus pixel and a secondfocus pixel may operate as a right focus pixel. In other examples, focuspixels may be arranged in an up-down pattern where a first focus pixelmay operate as an upper focus pixel and a second focus pixel may operateas a lower focus pixel.

In some examples, a configurable pixel may be dynamically adjusted tooperate as an imaging pixel or as a focus pixel based on an analysis ofa scene. For example, disclosed techniques may set a configurable pixelto operate as an imaging pixel for a first frame of a scene and may setthe configurable pixel to operate as a focus pixel for a subsequentframe of the scene. In some examples, disclosed techniques may set afirst focus orientation for a configurable pixel operating as a focuspixel for a frame and may set a second focus orientation for theconfigurable pixel for a subsequent frame of the scene.

Example techniques disclosed herein include receiving, at a processor,first image data from an image sensor coupled to the processor anddetermining at least one region of interest or region of non-interestbased on the first image data. As used herein, a “region of interest” ora “region of non-interest” may be generally referred to as an“identified region.” Disclosed techniques may then apply one or morescene analysis techniques on the first image data and identifiedregion(s) to determine how to adjust pixel configurations of the imagesensor. Disclosed techniques may then adjust a density of focus pixelswithin the image sensor that are associated with an identified regionbased on the determined pixel configurations. For example, disclosedtechniques may increase or decrease the quantity of focus pixelsassociated with the identified region to adjust the density of focuspixels within the image sensor.

In some examples, the one or more scene analysis techniques may includeperforming object detection and/or recognition techniques. For example,disclosed techniques may process the first image data and detect and/orrecognize one or more objects within an identified region. In someexamples, disclosed techniques may increase the density of focus pixelswithin the image sensor that are associated with the identified regionbased on the detected and/or recognized object. In some examples,disclosed techniques may additionally or alternatively decrease thedensity of focus pixels within the image sensor that are associated withthe identified region based on the detected and/or recognized object.For example, a scene may include a first identified region including aface, a second identified region including a microphone, and a thirdidentified region including a cup. In some examples, disclosedtechniques may set the density of focus pixels associated with the firstidentified region to a first density level, may set the density of focuspixels associated with the second identified region to a second densitylevel that is less dense than the first density level, and may set thedensity of focus pixels associated with the third identified region to athird density level that is less than the first density level and thesecond density level. In some examples, the first density level and thesecond density level may correspond to increasing the density of focuspixels within the image sensor that are associated with respectiveidentified regions and the third density level may correspond todecreasing the density of focus pixels within the image sensor that areassociated with the third identified region.

In some examples, disclosed techniques may perform focus dataorientation detection techniques based on an analysis of a scene. Forexample, in some instances, textures or edges within a region ofinterest may be oriented in a particular direction, such as a horizontaldirection or a vertical direction. Moreover, the corresponding focuspixels associated with the region of interest may be configured in afocus orientation such that the texture or edge may not be detected bythe generated focus data. For example, focus pixels arranged with aleft-right focus orientation may not detect a horizontal texture as thephase detection (PD) analysis of the focus data may be unable todetermine an offset between the left image and the right image generatedby the left focus pixels and the right focus pixels, respectively.

Example techniques disclosed herein set a first (or default) focusorientation for the focus pixels within the image sensor that areassociated with a region of interest. Disclosed techniques then performa PD analysis on image data generated by the image sensor while thefocus pixels are set in the first focus orientation. If a confidencelevel output by the PD analysis does not satisfy a confidence threshold,disclosed techniques may then set a second focus orientation for thefocus pixels associated with the region of interest, where the secondfocus orientation is different than the first focus orientation. Forexample, disclosed techniques may change the focus orientation of focuspixels associated with an identified region from a left-right focusorientation to an up-down focus orientation. However, if the confidencelevel satisfies the confidence threshold, disclosed techniques may thenmaintain the first focus orientation of the focus pixels. It should beappreciated that in some examples, the focus pixels associated withdifferent regions of interests may be associated with differentrespective focus orientations. Additionally, it should be appreciatedthat in some examples, the respective focus orientation of individualfocus pixels associated with a region of interest may vary.

The confidence level indicates the likelihood that the focus valuecomputed for the image is correct. The confidence level (sometimesreferred to as a “confidence measurement”) can be determined based on aspectral analysis of the image data corresponding to an edge, forexample. Sharper edges have high contrast between adjacent pixels, andthe spectral analysis shows that sharper edges have non-zerocoefficients for higher order frequency components. The confidence levelin an image as a function of focal length generally tracks the focusvalue as a function of focal length (lens position). The confidencelevel often has a peak at the focal length where the image is in focus,and falls off rapidly as the lens position moves away from the focusedposition. Low light levels or the presence of high-frequency patterns(e.g., closely spaced parallel lines) can reduce the confidence levelfor a focus value. When the image is severely out of focus, autofocuscan select a lens position having a local maximum contrast value, so theconfidence level is low. In most circumstances, with adequate lightingthe confidence level has a peak value when the focus value is accurate.The confidence value can be based on a variety of statistical measuressuch as mean, variance, standard deviation or other measures. In oneexample, if statistical analysis reveals that selected lens positionsfor the near or far focus bound include lens positions situated atsubstantial distances from one another, the confidence value may be low.

In some examples, disclosed techniques may perform saturation detectiontechniques based on an analysis of a scene. For example, in someinstances, a region of interest may include one or more saturated areas.As used herein, an area is saturated when the digital value output bythe image sensor for the corresponding pixels is greater than asaturation threshold. In some examples, the saturation threshold may bea predetermined value. In some examples, the saturation threshold may bea percentage. Information provided by focus pixels corresponding to thesaturated area may not be beneficial for autofocusing purposes as theprovided information may not be used for comparing images anddetermining an offset. Example techniques disclosed herein providetechniques for identifying a saturated area within an area of interest.Example techniques may then set the density of focus pixels within theimage sensor that are corresponding to the saturated area to a firstdensity level and may set the density of focus pixels that arecorresponding to the remaining area of the region of interest to asecond density level that is greater than the first density level. Insome examples, disclosed techniques may set the first density level tozero such that the configurable pixels corresponding to the saturatedarea are configured to operate as imaging pixels.

In some examples, disclosed techniques may perform edge detectiontechniques based on an analysis of a scene. In low-light environments,the image data generated by the image sensor may include a lowsignal-to-noise ratio (SNR) and/or high noise. To improve the quality ofsuch images, some examples apply de-noising techniques to the generatedimage data. For example, for a selected pixel, binning includesaveraging the value of two or more nearby pixels (referred tocollectively herein as “binning pixels”) and using the averaged value asthe value for the selected pixel. However, in some examples, an edge mayextend between the binning pixels. For example, a region of interest mayinclude a screen of a monitor (e.g., a first object) and a chassis ofthe monitor (e.g., a second object). As used herein, an “edge” refers tothe boundary created by two or more objects (e.g., the boundary betweenthe screen of the monitor and the chassis of the monitor). In some suchexamples, the binning pixels may include a pixel at a locationcorresponding to the first object and a pixel at a locationcorresponding to the second object (e.g., pixels corresponding todifferent objects). Binning or averaging the values of these pixels mayskew the average value for a selected pixel, resulting in an artifactand/or an incorrect representation of the image.

Example techniques disclosed herein may apply edge detection techniquesto detect an edge within a region of interest. If an edge is detected,disclosed techniques may then partition the region of interest andcorresponding pixels based on the detected edge. Example techniques maythen increase the quantity of focus pixels within the image sensor thatare associated with the respective region of interest portions. Byincreasing the quantity of focus pixels associated with the respectiveregions of interest portions, disclosed techniques may facilitate usingthe focus pixels associated with the respective region of interestportions to perform binning for same respective objects, therebyreducing (or avoiding) the performing of binning of focus pixels acrossthe edge and for two or more objects.

FIG. 2 is a block diagram that illustrates an example content generationsystem 200 configured to implement one or more techniques of thisdisclosure. The content generation system 200 includes a device 204. Asdescribed herein, a device, such as the device 204, may refer to anydevice, apparatus, or system configured to perform one or moretechniques described herein. For example, a device may be a server, abase station, user equipment, a client device, a station, an accesspoint, a computer (e.g., a personal computer, a desktop computer, alaptop computer, a tablet computer, a computer workstation, or amainframe computer), an end product, an apparatus, a phone, a smartphone, a server, a video game platform or console, a handheld device(e.g., a portable video game device or a personal digital assistant(PDA)), a wearable computing device (e.g., a smart watch, an augmentedreality device, or a virtual reality device), a non-wearable device, adisplay or display device, a television, a television set-top box, anintermediate network device, a digital media player, a video streamingdevice, a content streaming device, an in-car computer, any mobiledevice, any device configured to generate graphical content, or anydevice configured to perform one or more techniques described herein.Processes herein may be described as performed by a particular component(e.g., an ISP), but, in further embodiments, can be performed usingother components (e.g., an application processor or a CPU), consistentwith disclosed embodiments.

The device 204 may include one or more components or circuits forperforming various functions described herein. In some examples, one ormore components of the device 204 may be components of an SOC. Thedevice 204 may include one or more components configured to perform oneor more techniques of this disclosure. In the example shown, the device204 includes a processing unit 220, a memory 224, and an optical system250. In some examples, the device 204 can include a number of additionalor alternative components, such as a communication interface 226, atransceiver 232, a receiver 228, a transmitter 230, and a display client231.

In the illustrated example, the processing unit 220 includes an internalmemory 221. The processing unit 220 may be configured to perform imageprocessing, such as in image processing pipeline 209. Exampleimplementations of the image processing pipeline 209 may facilitateimage capture functions. In some examples, the processing unit 220 mayadditionally or alternatively be configured to perform graphicsprocessing, such as in a graphics processing pipeline and/ornon-graphics processing, such as in a compute processing pipeline.Example implementations of the compute processing pipeline mayfacilitate performing general-purpose operations or non-graphicaloperations, such as machine learning operations and/or artificialintelligence operations.

In some examples, the processing unit 220 includes an ISP (orapplication processor) configured to implement the image processingpipeline 209. The ISP may facilitate controlling image capturefunctions, such as autofocus, auto-white balance, and/or auto-exposure.In some examples, the ISP may also facilitate performing post-processingfunctions, such as depth mapping and/or Bokeh effect. In some examples,the ISP may also facilitate performing cropping, scaling (e.g., to adifferent resolution), image stitching, image format conversion, colorinterpolation, color processing, image filtering (e.g., spatial imagefiltering), lens artifact or defect correction, sharpening, or the like.

In some examples, the processing unit 220 may include a displayprocessor to perform one or more display processing techniques on one ormore frames generated by the processing unit 220 before presentment ofthe generated frame(s) by the display client 231. For example, thedisplay processor may be configured to perform one or more displayprocessing techniques on one or more frames generated by the processingunit 220. The display processor may output image data to the displayclient 231 according to an interface protocol, such as, for example, theMIPI DSI (Mobile Industry Processor Interface, Display SerialInterface).

The display client 231 may be configured to display or otherwise presentframes processed by the processing unit 220 (and/or the displayprocessor). In some examples, the display client 231 may include one ormore of: a liquid crystal display (LCD), a plasma display, an organiclight emitting diode (OLED) display, a projection display device, anaugmented reality display device, a virtual reality display device, ahead-mounted display, or any other type of display device.

Reference to the display client 231 may refer to one or more displays.For example, the display client 231 may include a single display ormultiple displays. The display client 231 may include a first displayand a second display. In further examples, the results of the imageprocessing may not be displayed on the device. For example, thedisplay(s) may not receive any frames for presentment thereon. Instead,the frames or image processing results may be transferred to anotherdevice. In some examples, the transferring of the frames or imageprocessing results to another device can be referred to assplit-rendering.

As disclosed above, the display client 231 may be configured inaccordance with MIPI DSI standards. The MIPI DSI standards support avideo mode and a command mode. In examples in which the display client231 is operating in the video mode, the processing unit 220 (and/or thedisplay processor) may continuously refresh the graphical content of thedisplay client 231. For example, the entire graphical content of a framemay be refreshed per refresh cycle (e.g., line-by-line).

In examples in which the display client 231 is operating in the commandmode, the processing unit 220 (and/or the display processor) may writethe graphical content of a frame to a buffer. In some examples, thedisplay client 231 may include the buffer and, thus, the buffer mayrepresent memory local to the display client 231. In some such examples,the processing unit 220 (and/or the display processor) may notcontinuously refresh the graphical content of the display client 231.Instead, the processing unit 220 (and/or the display processor) may usea vertical synchronization (Vsync) pulse to coordinate rendering andconsuming of graphical content at the buffer. For example, when a Vsyncpulse is generated, the processing unit 220 (and/or the displayprocessor) may output new graphical content to the buffer. Thus, thegenerating of the Vsync pulse may indicate when current graphicalcontent at the buffer has been rendered.

Memory external to the processing unit 220, such as memory 224, may beaccessible to the processing unit 220, the display client 231, and/orthe communication interface 226. For example, the processing unit 220may be configured to read from and/or write to external memory, such asthe memory 224. The processing unit 220, the display client 231, and/orthe communication interface 226 may be communicatively coupled to thememory 224 over a bus. In some examples, the processing unit 220, thememory 224, the communication interface 226, and/or the display client231 may be communicatively coupled to each other over the bus or adifferent connection.

In some examples, the device 204 may include a content encoder/decoderconfigured to receive graphical and/or display content from any source,such as the memory 224 and/or the communication interface 226. Thememory 224 may be configured to store received encoded content ordecoded content. In some examples, the content encoder/decoder may beconfigured to receive encoded content or decoded content (e.g., from thememory 224 and/or the communication interface 226) in the form ofencoded pixel data or decoded pixel data. In some examples, the contentencoder/decoder may be configured to encode or decode any content.

The internal memory 221 and/or the memory 224 may include one or morevolatile or non-volatile memories or storage devices. In some examples,the internal memory 221 and/or the memory 224 may include RAM, SRAM,DRAM, erasable programmable ROM (EPROM), electrically erasableprogrammable ROM (EEPROM), flash memory, a magnetic data media or anoptical storage media, or any other type of memory.

The internal memory 221 and/or the memory 224 may be a non-transitorystorage medium according to some examples. The term “non-transitory” mayindicate that the storage medium is not embodied in a carrier wave or apropagated signal. However, the term “non-transitory” should not beinterpreted to mean that the internal memory 221 and/or the memory 224is non-movable or that its contents are static. As one example, thememory 224 may be removed from the device 204 and moved to anotherdevice. As another example, the memory 224 may not be removable from thedevice 204.

The processing unit 220 may be a CPU, an application processor, an ISP,a GPU, a general purpose GPU (GPGPU), a DPU, a display processor, or anyother processing unit that may be configured to perform imageprocessing. In some examples, the processing unit 220 may be integratedinto a motherboard of the device 204. In some examples, the processingunit 220 may be present on a graphics card that is installed in a portin a motherboard of the device 204, or may be otherwise incorporatedwithin a peripheral device configured to interoperate with the device204. The processing unit 220 may include one or more processors, such asone or more microprocessors, CPUs, application processors, GPUs, DPUs,display processors, ISPs, application specific integrated circuits(ASICs), field programmable gate arrays (FPGAs), arithmetic logic units(ALUs), digital signal processors (DSPs), discrete logic, software,hardware, firmware, other equivalent integrated or discrete logiccircuitry, or any combinations thereof. If the techniques areimplemented partially in software, the processing unit 220 may storeinstructions for the software in a suitable, non-transitorycomputer-readable storage medium (e.g., the internal memory 221), andmay execute the instructions in hardware using one or more processors toperform the techniques of this disclosure. Any of the foregoing,including hardware, software, a combination of hardware and software,etc., may be considered to be one or more processors.

In the illustrated example, the device 204 includes a communicationinterface 226. The communication interface 226 may include a receiver228 and a transmitter 230. The receiver 228 may be configured to performany receiving function described herein with respect to the device 204.Additionally, the receiver 228 may be configured to receive information(e.g., eye or head position information, rendering commands, or locationinformation) from another device. The transmitter 230 may be configuredto perform any transmitting function described herein with respect tothe device 204. For example, the transmitter 230 may be configured totransmit information to another device, which may include a request forcontent. The receiver 228 and the transmitter 230 may be combined into atransceiver 232. In such examples, the transceiver 232 may be configuredto perform any receiving function and/or transmitting function describedherein with respect to the device 204.

In the illustrated example, the device 204 includes the optical system250 in communication with the processing unit 220. The optical system250 includes a lens assembly 252, an image sensor 254, and a chargecomponent 256. The lens assembly 252 may facilitate focusing incominglight onto the pixels of the image sensor 254. It should be appreciatedthat the lens assembly 252 may include any number of optical elements.In some examples, the processing unit 220 may be configured to shift thelens assembly 252 to adjust the focus of the light received on the imagesensor 254. It should be appreciated that the optical system 250 mayinclude one or more additional optical components mounted inside ahousing of the device 204 and/or positioned on the housing or the lensassembly 252. For example, the additional optical components may includea motion sensor (e.g., an accelerometer, a gyroscope, etc.), apertures,shutters, mirrors, filters, coatings, etc.

The example image sensor 254 may be a complementary metal oxidesemiconductor (CMOS) imaging sensor or a charge-coupled device (CCD)sensor. However, it should be appreciated that in other examples, theimage sensor may be any suitable sensor including pixels for capturingimage data. The example image sensor 254 of FIG. 2 includes a pluralityof pixels for capturing image data. The image sensor 254 may beconfigured to digitize the amount of light received by each pixel. Forexample, when the received light is expressed as a ten-bit digitalvalue, a low value (e.g., 100) may indicate a low amount of light, and ahigh value (e.g., 1023) may indicate a high amount of light. Theplurality of pixels may also include a set of pixels that areconfigurable to operate as focus pixels or imaging pixels. In someexamples, the set of configurable pixels may comprise one to all of theplurality of pixels of the image sensor 254.

FIG. 3 illustrates a side view of a configurable pixel 300 of the imagesensor 254 of FIG. 2 , in accordance with techniques disclosed herein.The configurable pixel 300 includes three components in common with animaging pixel, including a microlens 310, a color filter 320, and aphotodiode 330. The microlens 310 may be configured to focus receivedlight onto the photodiode 330. The color filter 320 may be red, green,or blue (R, G, or B) arranged in a Bayer pattern. In other examples, thecolor filter 320 may be arranged in a cyan, yellow, green, and magentapattern, a red, green, blue, and emerald pattern, a cyan, magentayellow, and white pattern, a red, green, blue, and white pattern, orother pattern. The photodiode 330 of FIG. 3 is positioned within asubstrate 340, such as a silicon substrate.

The configurable pixel 300 of FIG. 3 includes an opacity transitioningmaterial 350 positioned between the color filter 320 and the photodiode330. The opacity transitioning material 350 may be an electricallycontrolled material, such as a liquid crystal, that may appeartransparent or opaque. When the opacity transitioning material 350 istransparent, light may be received by the photodiode 330. When theopacity transitioning material 350 is opaque, light is blocked frombeing received by the photodiode 330.

In the illustrated example of FIG. 3 , the opacity transitioningmaterial 350 is divided into a first opacity transitioning materialportion 350 a and a second opacity transitioning material portion 350 b,and each portion 350 a, 350 b may be independently controlled. When nocharge (or current or voltage) is applied to the opacity transitioningmaterial 350, the opacity transitioning material 350 appears transparentand the configurable pixel 300 may operate as an imaging pixel, such asthe red/green/blue pixels 104 of FIG. 1 . When a charge (or current orvoltage) is applied to the opacity transitioning material 350, therespective portion 350 a, 350 b becomes opaque, which generates asimilar effect as using a metal shielding to block light (as shown inFIG. 1 ), and the configurable pixel may operate as a focus pixel, suchas the focus pixels 106 of FIG. 1 . Example techniques for applying thecharge (or current or voltage) are described below in connection withthe charge component 256 of FIG. 2 .

Accordingly, the configurable pixel 300 may be configured to operate asa focus pixel or an imaging pixel based on whether a charge is appliedto the opacity transitioning material 350. Moreover, the density offocus pixels associated with a region of interest of an image may beadjusted by applying (or not applying) a charge to respective ones ofthe configurable pixels 300 of the image sensor 254.

In some examples, the plurality of pixels of the image sensor 254 may beconfigurable pixels. In some such examples, each pixel of the imagesensor 254 may be configured to operate as a focus pixel and/or animaging pixel.

In some examples, the set of configurable pixels may be less than theplurality of pixels of the image sensor 254. For example, one or more ofthe pixels of the image sensor 254 may be dedicated imaging pixels andthe remaining pixels may be configurable pixels. In some such examples,the dedicated imaging pixels may be positioned along the periphery ofthe image sensor 254. It should be appreciated that the ratio ofdedicated imaging pixels to configurable pixels may vary.

In some examples, the opacity transitioning material 350 may be dividedinto two independently controllable portions (as shown in FIG. 3 ). FIG.4A illustrates a top view of a configurable pixel 400 including a first(or left) opacity transitioning material portion 410 and a second (orright) opacity transitioning material portion 420. In the illustratedexample of FIG. 4A, the portions 410, 420 are positioned so that theconfigurable pixel 400 may operate as a left-right focus pixel. Forexample, a charge may be applied to the left opacity transitioningmaterial portion 410 and the configurable pixel 400 may operate as a“right focus pixel,” because light entering the right opacitytransitioning material portion 420 may reach the photodiode. Similarly,when a charge is applied to the right opacity transitioning materialportion 420, the configurable pixel 400 may operate as a “left focuspixel,” because light entering the left opacity transitioning materialportion 410 may reach the photodiode. Aspects of the portions 410, 420may be implemented by the portions 350 of FIG. 3 .

FIG. 4B illustrates a top view of a configurable pixel 450 including afirst (or upper) opacity transitioning material portion 460 and a second(or lower) opacity transitioning material portion 470. In theillustrated example of FIG. 4B, the portions 460, 470 are positioned sothat the configurable pixel 400 may operate as an up-down focus pixel.For example, a charge may be applied to the upper opacity transitioningmaterial portion 460 and the configurable pixel 450 may operate as a“down focus pixel,” because light entering the lower opacitytransitioning material portion 470 may reach the photodiode. Similarly,when a charge is applied to the lower opacity transitioning materialportion 470, the configurable pixel 450 may operate as an “up focuspixel,” because light entering the upper opacity transitioning materialportion 460 may reach the photodiode. Aspects of the portions 460, 470may be implemented by the portions 350 of FIG. 3 .

FIGS. 5A to 5F illustrate top views of focus orientations forconfigurable pixels in which the opacity transitioning material isdivided into four independently-controllable portions 502, 504, 506,508. Aspects of the portions 502, 504, 506, 508 may be implemented bythe portions 350 of FIG. 3 , the portions 410, 420 of FIG. 4A, and/orthe portions 460, 470 of FIG. 4B. In some examples, the focusorientation of the configurable pixel may be determined based on whichportions of the opacity transitioning materials a charge is applied.FIGS. 5A and 5B illustrate configurable pixels with a left-right focusorientation. For example, FIG. 5A illustrates a configurable pixel 500configured to operate as a “right focus pixel,” because a charge isapplied to a first opacity transitioning material portion 502 and athird opacity transitioning material portion 506, and a charge is notapplied to a second opacity transitioning material portion 504 and afourth opacity transitioning material portion 508. FIG. 5B illustrates aconfigurable pixel 510 configured to operate as a “left focus pixel,”because a charge is applied to the second opacity transitioning materialportion 504 and the fourth opacity transitioning material portion 508,and a charge is not applied to the first opacity transitioning materialportion 502 and the third opacity transitioning material portion 506.

FIGS. 5C and 5D illustrate configurable pixels with an up-down focusorientation. For example, FIG. 5C illustrates a configurable pixel 520configured to operate as an “upper focus pixel,” because a charge isapplied to the third opacity transitioning material portion 506 and thefourth opacity transitioning material portion 508, and a charge is notapplied to the first opacity transitioning material portion 502 and thesecond opacity transitioning material portion 504. FIG. 5D illustrates aconfigurable pixel 530 configured to operate as a “lower focus pixel,”because a charge is applied to the first opacity transitioning materialportion 502 and the second opacity transitioning material portion 504,and a charge is not applied to the third opacity transitioning materialportion 506 and the fourth opacity transitioning material portion 508.

FIGS. 5E and 5F illustrate configurable pixels with a diagonal focusorientation. For example, FIG. 5E illustrates a configurable pixel 540configured to operate as an “upper-left, lower-right focus pixel,”because a charge is applied to the second opacity transitioning materialportion 504 and the third opacity transitioning material portion 506,and a charge is not applied to the first opacity transitioning materialportion 502 and the fourth opacity transitioning material portion 508.FIG. 5F illustrates a configurable pixel 550 configured to operate as an“upper-right, lower-left focus pixel,” because a charge is applied tothe first opacity transitioning material portion 502 and the fourthopacity transitioning material portion 508, and a charge is not appliedto the second opacity transitioning material portion 504 and the thirdopacity transitioning material portion 506.

Although the example configurable pixels of FIGS. 4A and 4B illustratethe opacity transitioning material arranged into two portions and theexample configurable pixels of FIGS. 5A to 5F illustrate the opacitytransitioning material arranged into four portions, it should beappreciated that other examples may include additional or alternativequantity and/or arrangement of portions. Moreover, it should beappreciated that while the illustrated examples of FIGS. 5A to 5Finclude setting two of the four opacity transitioning material portionsto opaque, in other examples, any suitable quantity of the opacitytransitioning material portions may be configured to be opaque to enablethe respective pixel to operate as a focus pixel.

Furthermore, it should be appreciated that in some examples, differentconfigurable pixels may be configured to operate as focus pixels havingdifferent focus orientations. For example, a first subset ofconfigurable pixels may be configured to operate as focus pixels with aleft-right focus orientation (as shown in FIGS. 5A and 5B), a secondsubset of configurable pixels may be configured to operate as focuspixels with an up-down focus orientation (as shown in FIGS. 5C and 5D),and/or a third subset of configurable pixels may be configured tooperate as focus pixels with a diagonal focus orientation (as shown inFIGS. 5E and 5F).

Referring again to FIG. 2 , the charge component 256 may be coupled toportions of the opacity transitioning material of the selectedconfigurable pixels of the set of the configurable pixels, for example,via a wire. The charge component 256 may be configured to apply a charge(or current or voltage) to portions of the opacity transitioningmaterial of selected configurable pixels of the set of the configurablepixels (e.g., via the wire) to cause the selected configurable pixels tooperate as focus pixels. Additionally, the charge component 256 may beconfigured to not apply a charge (or current or voltage) to thenon-selected configurable pixels of the set of the configurable pixels(e.g., via the wire) to cause the non-selected configurable pixels tooperate as imaging pixels. Although the above description provides anexample in which the charge component 256 is coupled to portions of theopacity transitioning material via a wire, it should be appreciated thatother examples may use additional or alternative techniques for couplingthe charge component 256 to the portions of the opacity transitioningmaterial. For example, the charge component 256 may be coupled to theportions of the opacity transitioning material via an electrode.

Referring still to FIG. 2 , in some aspects, the processing unit 220 maybe configured to operate one or more techniques disclosed herein. Forexample, the processing unit 220 may include a determination component298 configured to receive first image data of a first frame of a scene.The determination component 298 may be configured to determine at leastone region of interest or region of non-interest of the first frame. Thedetermination component 298 may be configured to select, based on thedetermined at least one region of interest or region of non-interest, asubset of the set of configurable pixels to be focus pixels. Thedetermination component 298 may also be configured to cause the selectedsubset of the set of configurable pixels to operate as focus pixels.

FIG. 6 illustrates a block diagram of an image processor 600, inaccordance with aspects of this disclosure. One or more aspects of theimage processor 600 may be implemented by the processing unit 220 ofFIG. 2 . The example image processor 600 may include one or moreprocessors that are configured to execute an object handling component610, an orientation handling component 620, a saturation handlingcomponent 630, and an edge handling component 640.

In the illustrated example, the image processor 600 is configured toreceive image data for a frame. For example, the image processor 600 mayreceive image data for a frame of a scene from the image sensor 254 ofFIG. 2 . In some examples, the image data may include no focus data. Forexample, the configurable pixels of the image sensor 254 may beconfigured to operate as imaging pixels when the image sensor 254generates the image data. In some examples, the image data may includefocus data. For example, a subset of the configurable pixels of theimage sensor 254 may be configured to operate as focus pixels when theimage sensor 254 generates the image data. In some examples, the subsetof the configurable pixels configured to operate as focus pixels may bea default subset of the configurable pixels. For example, the defaultsubset of the configurable pixels may be a predetermined quantity and/orpattern of focus pixels that is not based on scene analysis techniquesperformed on previously received image data (e.g., for a previousframe). In some examples, the subset of the configurable pixelsconfigured to operate as focus pixels may be based on the performing ofthe one or more scene analysis techniques on previously received imagedata. In some examples, the focus data may be used to adjust the focusof the device 204.

The example image processor 600 may also be configured to perform one ormore scene analysis techniques on the received image data, and select asubset of the configurable pixels of the image sensor 254 to operate asfocus pixels based on the performing of the one or more scene analysistechniques. The image processor 600 may cause the selected subset ofconfigurable pixels to operate as focus pixels by instructing the chargecomponent 256 to apply a charge to the respective opacity transitioningmaterial portion(s) for the selected subset of configurable pixels.

The example image processor 600 may also be configured to identifyregions of interest and/or regions of non-interest in the image based onan analysis of the image data. For example, the image processor 600 mayidentify regions utilizing artificial intelligence mechanisms and/ormachine learning mechanisms. In some examples, the image processor 600may receive information indicating a region of interest or a region ofnon-interest. For example, the image processor 600 may receive userinput via an input interface. In some such examples, the image processor600 may determine at least one region of interest and/or region ofnon-interest based on the received user input.

In some examples, the image processor 600 may be configured to adjustthe density of focus pixels within the image sensor that are associatedwith an identified region based on the performing of the one or morescene analysis techniques. For example, the image processor 600 may beconfigured to increase or decrease the quantity of focus pixelsassociated with an identified region. In some examples, the imageprocessor 600 may be configured to adjust the quantity of focus pixelsassociated with a sub-region of the identified region. For example, theimage processor 600 may identify a region (or area) of non-interestwithin a region of interest. In some such examples, the image processor600 may increase the quantity of focus pixels of the image sensor thatare associated with the region of interest and also decrease thequantity of focus pixels associated with the identified region ofnon-interest.

In the illustrated example of FIG. 6 , the image processor 600 isconfigured to include the object handling component 610 to facilitateimplementing object detection and/or recognition techniques. In someexamples, an image may include one or more identified regions. In somesuch examples, the object handling component 610 may be configured todetermine whether an identified region is a region of interest or aregion of non-interest. For example, the object handling component 610may be configured to implement one or more object detection and/orrecognition mechanisms to identify (e.g., detect and/or recognize) anobject in the image.

FIG. 7 depicts an image 700 including a person 710 wearing a hat 720 inthe foreground and a wall 730 in the background. In the illustratedexample, the object handling component 610 of FIG. 6 may identify afirst region 712 corresponding to the face of the person 710, a secondregion 722 corresponding to the hat 720, and a third region 732corresponding to the wall 730. The object handling component 610 mayclassify each identified region 712, 722, 732 based on the objectdetected and/or recognized within the respective region. For example,the object handling component 610 may classify the first region 712corresponding to the face of the person 710 and the second region 722corresponding to the hat 720 as regions of interest. The object handlingcomponent 610 may classify the third region 732 corresponding to thewall 730 as a region of non-interest. In some examples, the objecthandling component 610 may be configured to classify the region based ona target item or an item of interest, such as a face or person.

In some examples, the object handling component 610 may be configured tosub-classify the identified regions based on the detected and/orrecognized objects. For example, the object handling component 610 maybe configured to classify the face of the person 710 as an object offirst-level importance, classify the hat 720 as an object ofsecond-level importance, and classify the wall 730 as an object ofthird-level importance. In some such examples, the second-level ofimportance may be less important than the first-level of importance andmay be more important than the third-level of importance.

The object handling component 610 may also be configured to determinehow to adjust the density of focus pixels corresponding to each of theidentified regions. For example, the object handling component 610 maydetermine to increase the density of focus pixels associated withregions of interest and/or to decrease the density of focus pixelsassociated with regions of non-interest. In some examples, the densityof focus pixels associated with a region of non-interest may bedecreased such that the configurable pixels corresponding to the regionof non-interest are configured to operate as imaging pixels. In someexamples, the object handling component 610 may determine to adjust thedensity of focus pixels based on the different levels of importance. Forexample, regions of first-level importance may be associated with afirst density-level, regions of second-level importance may beassociated with a second density-level, and regions of third-levelimportance may be associated with a third density-level. In some suchexamples, the second density-level may be less dense than the firstdensity-level and may be more dense than the third density level.

It should be appreciated that areas of the image that do not correspondto the identified regions may be associated with a backgrounddensity-level. For example, the area of the image 700 corresponding tothe hair or shirt of the person 710 may be associated with thebackground density-level. In some examples, one or more of thedensity-levels may be of greater density than the backgrounddensity-level and/or may be of less density than the backgrounddensity-level. For example, in some examples, the first density-level,the second density-level, and the third density-level may be greaterthan the background density-level. In some examples, the density-levelfor regions of interest (e.g., the first region 712 and the secondregion 722) may be of greater density than the background density-leveland the density-level for regions of non-interest (e.g., the thirdregion 732) may be less dense than the background density-level.

In the illustrated example of FIG. 6 , the image processor 600 isconfigured to include the orientation handling component 620 tofacilitate implementing focus data orientation detection techniques. Insome examples, an identified region may include a texture or edge thatis oriented in a particular direction, such as a horizontal texture or avertical texture. In some such examples, if the focus pixelscorresponding to the identified region have a same focus orientation asthe texture or edge, then the focus data generated by the respectivefocus pixels may not be useful for PDAF purposes.

In some examples, the orientation handling component 620 may beconfigured to perform a confidence test based on the focus data receivedfor image data. For example, the confidence test may measure noiseassociated with the focus data along the focus orientation of therespective focus pixels. FIG. 8 depicts a region of interest 800including a plurality of horizontal textures 810. It should beappreciated that if the focus pixels corresponding to the region ofinterest 800 have a left-right focus orientation, focus data generatedby the focus pixels may not provide useful information for PDAFpurposes. For example, the orientation handling component 620 may beunable to determine the offset between a left image and a right imagegenerated by left focus pixels and right focus pixels, respectively.

In the illustrated example, the orientation handling component 620 maybe configured to set a first focus orientation for focus pixelsassociated with an identified region. For example, the first focusorientation may be a default focus orientation, such as a left-rightfocus orientation (as shown in FIGS. 5A and 5B). The orientationhandling component 620 may perform the confidence test based on imagedata received from the focus pixels having the first focus orientation.

In some examples, if the confidence level generated by the confidencetest satisfies a confidence threshold, the orientation handlingcomponent 620 may determine that the current focus orientation of thefocus pixels associated with the identified region is capable ofproviding useful information for PDAF purposes. However, if theconfidence level does not satisfy the confidence threshold, theorientation handling component 620 may determine to change the focusorientation of the focus pixels associated with the identified region.For example, the orientation handling component 620 may determine tochange the focus orientation to the up-down focus orientation (as shownin FIGS. 5C and 5D) or the diagonal focus orientation (as shown in FIGS.5E and 5F).

In some examples, the confidence level may satisfy the confidencethreshold when the confidence level is greater than or equal to theconfidence threshold. In some examples, the confidence level may satisfythe confidence threshold when the confidence level is greater than theconfidence threshold. However, it should be appreciated that in someexamples, the confidence level may satisfy the confidence threshold whenthe confidence level is less than or equal to the confidence thresholdor when the confidence level is less than the confidence threshold.

In some examples, the confidence level may be a numerical value, such asa number between zero and one hundred. In some such examples, theconfidence threshold may be a numerical value between zero and onehundred. In some examples, the confidence level may be binary valuecorresponding to a “confident” or “non-confident” level. In some suchexamples, the confidence threshold may be a value corresponding to“confident.”

It should be appreciated that by adjusting the focus orientation offocus pixels of the image sensor that are associated with an identifiedregion when the confidence level does not satisfy the confidencethreshold, the orientation handling component 620 facilitates increasingthe likelihood of receiving useful information for PDAF purposes fromfocus data generated during a subsequent frame.

In the illustrated example of FIG. 6 , the image processor 600 isconfigured to include the saturation handling component 630 tofacilitate implementing saturation detection techniques. In some images,a region of interest may include a saturated area. Focus data generatedby focus pixels associated with a saturated area may not provide usefulinformation for PDAF purposes. For example, focus data generated byfocus pixels associated with the saturated area may not providesufficient detail to measure an offset between focus pixel pairs.

FIG. 9 depicts an example image 900 including an identified region ofinterest 910 corresponding to a car. The example saturation handlingcomponent 630 may be configured to apply saturated area detectiontechniques to identify one or more saturated areas in a frame. In someexamples, the saturation handling component 630 may identify a saturatedarea based on the digital values generated by the image sensor 254. Forexample, for a ten-bit digital value, the image sensor 254 may generatea digital value between 0 and 1023. In some such examples, the digitalvalue 1023 represents a maximum amount of light that the image sensor254 is capable of digitizing. In some examples, the saturation handlingcomponent 630 may be configured to apply a percentage of the maximumdigital value capable of being output by the image sensor 254 todetermine when an area is saturated.

In the illustrated example of FIG. 9 , the saturation handling component630 may identify a first saturated area 920 corresponding to a frontheadlight of a car and a second saturated area 930 corresponding to arear taillight of another car. As shown in FIG. 9 , the first saturatedarea 920 is within the identified region of interest 910.

In some examples, the saturation handling component 630 may determine toreduce the density of focus pixels of the image sensor that areassociated with the saturated areas 920, 930. For example, thesaturation handling component 630 may set the quantity of focus pixelsassociated with the saturated areas 920, 930 to zero focus pixels sothat the configurable pixels of the image sensor that are associatedwith the saturated areas 920, 930 are configured to operate as imagingpixels.

In some examples, the saturation handling component 630 may determinewhether a saturated area overlaps with an identified region of interest.For example, the saturation handling component 630 may determine thatthe identified region of interest 910 and the first saturated area 920overlap. In some such examples, the saturation handling component 630may determine to increase the density of focus pixels associated withthe identified region of interest 910 and may determine to reduce thedensity of focus pixels associated with the first saturated area 920.For example, the saturation handling component 630 may determine to setthe density-level of focus pixels associated with the identified regionof interest 910 and outside the first saturated to a first density thatis greater than the background density-level. The saturation handlingcomponent 630 may also determine to set the density-level of focuspixels associated with the first saturated area 920 to a second densitythat is less than the first density. In some examples, the seconddensity may correspond to zero focus pixels so that the configurablepixels associated with the first saturated area 920 are configured tooperate as imaging pixels. In some examples, the second density maycorrespond to a density-level that is greater than zero focus pixels andless than the background density-level.

It should be appreciated that by reducing the density-level of focuspixels associated with a saturated area, the saturation handlingcomponent 630 facilitates reducing the amount of focus data generatedthat may not be useful for PDAF purposes. Additionally, by setting theconfigurable pixels associated with a saturated area to operate asimaging pixels, the saturation handling component 630 may alsofacilitate improving the quality of the image.

In the illustrated example of FIG. 6 , the image processor 600 may beconfigured to include the edge handling component 640 to facilitateimplementing edge detection techniques. In some examples, a scene maycorrespond to a low-light environment. For example, the digital valuesgenerated by the image sensor 254 may be less than a low-lightenvironment threshold. For example, for a ten-bit digital value (rangingbetween 0 and 1023), a low-light environment may be indicated by digitalvalues that are less than 100. However, it should be appreciated thatadditional or alternative examples may use other threshold values.

In a low-light environment, the digital values generated by the imagesensor 254 may have low signal-to-noise ratios (SNR) and/or high noise.To improve the quality of the image, the image processor 600 may performde-noising techniques, such as binning (or averaging) values (or data)of a plurality of nearby focus pixels having a same focus orientation(referred to collectively as “binning pixels”).

FIG. 10 illustrates a portion of an image sensor 1000 including an array1002 of pixels 1004. In the illustrated example, the portion of theimage sensor 1000 corresponds to the configurable pixels operating aseither imaging pixels or focus pixels associated with a region ofinterest for a first frame. In the illustrated example, the array 1002includes a plurality of left focus pixels 1010 and a plurality of rightfocus pixels 1020. When performing binning, the image processor 600 mayaverage the digital values of focus pixels having a same focusorientation. For example, in the illustrated example of FIG. 10 , theimage processor 600 may average the digital values of a first left focuspixel 1010 a, a second left focus pixel 1010 b, a third left focus pixel1010 c, and a fourth left focus pixel 1010 d to determine the value ofthe second left focus pixel 1010 b. Similarly, the binning pixels fordetermining the value of a second right focus pixel 1020 b may include afirst right focus pixel 1020 a, the second right focus pixel 1020 b, athird right focus pixel 1020 c, and a fourth right focus pixel 1020 d.

In some examples, the region of interest may include an edge. An edgemay be determined by a change in luminance or a change in color. Forexample, the edge handling component 640 may be configured to detect anedge when the change in luminance and/or the change in color is greaterthan a threshold percentage. In some examples, the edge may correspondto a border between different objects. For example, referring to theexample image 700 of FIG. 7 , the edge handling component 640 may detectan edge within the second identified region 722 between the hat 720 andthe forehead of the face of the person 710. In some examples, the edgehandling component 640 may detect an edge between different areas of anobject. For example, the edge handling component 640 may detect an edgebetween a screen of a monitor and a chassis of the monitor.

In the illustrated example of FIG. 10 , an edge is detected within thefirst image associated with pixels between a first subset 1002 a ofpixels 1004 and a second subset 1002 b of pixels 1004. Thus, the pixels1004 within the first subset 1002 a may be associated with a firstsimilar luminance or a first similar color and the pixels 1004 withinthe second subset 1002 b may be associated with a second similarluminance or a second similar color.

However, it should be appreciated that when binning, determining thevalue of a focus pixel by averaging the digital value of focus pixelsacross the edge may skew the average value for the focus pixel, whichmay result in an artifact. For example, determining the value of thesecond left focus pixel 1010 b include averaging the digital value ofthe first left focus pixel 1010 a and the second left focus pixel 1010b, which are within the first subset 1002 a of pixels 1004 and areassociated with the first similar luminance or the first similar color,and the third left focus pixel 1010 c and the fourth left focus pixel1010 d, which are within the second subset 1002 b of pixels 1004 and areassociated with the second similar luminance or the second similarcolor.

To improve the performing of binning in low-light environments, the edgehandling component 640 is configured to detect one or more edges withina region of interest.

If the edge handling component 640 detects an edge within a region ofinterest, the edge handling component 640 is configured to increase thedensity of focus pixels associated with respective sub-regions of theregion of interest. For example, referring to the example of FIG. 10 ,the edge handling component 640 may be configured to increase thedensity of left focus pixels 1010 and right focus pixels 1020 within thefirst subset 1002 a of pixels 1004 of the image sensor 1000. The edgehandling component 640 may additionally or alternatively be configuredto increase the density of left focus pixels 1010 and right focus pixels1020 within the second subset 1002 b of pixels 1004 of the image sensor1000.

It should be appreciated that by increasing the density of focus pixelsassociated with the respective sub-regions of the region of interest,the edge handling component 640 may facilitate improving the quality ofbinning by reducing the likelihood of binning pixels extending across anedge. It should be appreciated that the binning may be performed on asubsequent frame of the scene.

In the illustrated example, after the object handling component 610, theorientation handling component 620, the saturation handling component630, and/or the edge handling component 640 perform their respectivescene analysis techniques and determine the changes, if any, to be madeto the configurable pixels of the image sensor 254, the image processor600 applies the respective pixel configurations for a subsequent frame.For example, a change in a pixel configuration may include changing aconfigurable pixel from operating as an imaging pixel to operating as afocus pixel (e.g., to increase the density of focus pixels associatedwith an identified region), changing a configurable pixel from operatingas a focus pixel to operating as an imaging pixel (e.g., to decrease thedensity of focus pixels associated with an identified region), and/orchanging a focus orientation of a focus pixel from a first focusorientation to a second focus orientation (e.g., to increase thelikelihood of receiving useful information for PDAF purposes from focusdata generated by focus pixel associated with a region including atexture).

FIGS. 11 to 15 illustrate example flowcharts of example methods inaccordance with one or more techniques disclosed herein. The methods maybe performed by an apparatus, such as the example device 204 of FIG. 2 ,and/or a component of the apparatus, such as the example processing unit220 of FIG. 2 and/or the image processor 600 of FIG. 6 . According tovarious aspects, one or more of the illustrated operations of themethods may be omitted, transposed, and/or contemporaneously performed.Optional aspects are illustrated with a dashed line. In the illustratedexamples of FIGS. 11 to 15 , the apparatus is configured to include animage sensor having a plurality of pixels including a set of pixelsconfigurable to operate as imaging pixels or focus pixels, such as theexample image sensor 254 of FIG. 2 .

FIG. 11 is a flowchart 1100 of a method employing a PDAF optical system,in accordance with one or more techniques disclosed herein. At 1102, theapparatus receives first image data of a first frame of a scene, asdescribed in connection with the examples in FIGS. 2 to 10 . Forexample, the processing unit 220 and/or the image processor 600 may beconfigured to receive the first image data from the image sensor 254. Itmay be appreciated that the configurable pixels of the image sensor 254may be configured to operate with a default configuration (e.g., asimaging pixels or as focus pixels having a first focus orientation) ormay be configured to operate with a previous pixel configuration (e.g.,a first subset of configurable pixels are configured to operate asimaging pixels, a second subset of configurable pixels are configured tooperate as focus pixels having a first focus orientation, a third subsetof configurable pixels are configured to operate as focus pixels havinga second focus orientation, etc.).

At 1104, the apparatus determines at least one region of interest orregion of non-interest of the first frame, as described in connectionwith the examples in FIGS. 2 to 10 . For example, the processing unit220 and/or the image processor 600 may be configured to determine the atleast one region of interest or region of non-interest of the firstframe. In some examples, the apparatus may determine the identifiedregion by applying object detection and/or recognition techniques. Insome examples, the apparatus may receive user input indicating the atleast one region of interest or region of non-interest and may thendetermine the region of interest or region of non-interest based on thereceived user input. It may be appreciated that in some examples, theregion of interest or the region of non-interest may comprise the fullframe.

At 1106, the apparatus may apply one or more scene analysis techniquesto the at least one region of interest or region of non-interest todetermine pixel configurations for the configurable pixels, as describedin connection with the examples in FIGS. 2 to 10 . For example, theprocessing unit 220 and/or the image processor 600 may be configured toapply the one or more scene analysis techniques to the at least oneregion of interest or region of non-interest to determine pixelconfigurations. In some examples, the apparatus may apply one or more ofthe object detection and/or recognition techniques, the focus dataorientation detection techniques, the saturation detection techniques,and the edge detection techniques. Aspects of the object detectionand/or recognition techniques are described below in connection withflowchart 1200 of FIG. 12 . Aspects of the focus data orientationdetection techniques are described below in connection with flowchart1300 of FIG. 13 . Aspects of the saturation detection techniques aredescribed below in connection with flowchart 1400 of FIG. 14 . Aspectsof the edge detection techniques are described below in connection withflowchart 1500 of FIG. 15 .

In some examples, the apparatus may determine to apply each of the fourexample scene analysis techniques. In some examples, the apparatus maydetermine to apply a subset of the example scene analysis techniques. Insome examples, the apparatus may apply one or more of the example sceneanalysis techniques for each frame of a scene. In some examples, theapparatus may conserve power and/or computational resources of theapparatus by skipping one or more frames on which one or more of theexample scene analysis techniques are applied. In some examples, theapparatus may perform an initial scene analysis (e.g., based on machinelearning and/or artificial intelligence) to determine whether aspects ofa scene changed between frames. In some such examples, the apparatus mayconserve power and/or computational resources of the apparatus byapplying the one or more scene analysis techniques when aspects of thescene changed between frames.

At 1108, the apparatus selects a subset of the set of configurablepixels of the image sensor to be focus pixels, as described inconnection with the examples in FIGS. 2 to 10 . For example, theprocessing unit 220 and/or the image processor 600 may be configured toselect the subset of the set of configurable pixels to be focus pixels.In some examples, the apparatus may use the pixel configurationsdetermined by the applying of the one or more scene analysis techniques(at 1106) to determine the subset of the set of configurable pixels tobe focus pixels. In some examples, the pixel configurations may alsoprovide focus orientations for the subset of configurable pixelsconfigured to operate as focus pixels.

At 1110, the apparatus causes the selected subset of the configurablepixels to operate as focus pixels, as described in connection with theexamples in FIGS. 2 to 10 . For example, the processing unit 220 and/orthe image processor 600 may be configured to cause the charge component256 to apply a charge (or current or voltage) to the opacitytransitioning material portions 350 of the subset of the configurablepixels of the image sensor 254 to cause the respective configurablepixels to operate as focus pixels. In some examples, the apparatus mayadditionally or alternatively change a focus orientation for one or moreof the focus pixels. In some examples, the apparatus may additionally oralternatively cause one or more of the configurable pixels to operate asimaging pixels.

At 1112, the apparatus may shift the lens assembly to adjust the focusof the received light on the image sensor, as described in connectionwith the examples in FIGS. 2 to 10 . For example, the processing unit220 and/or the image processor 600 may be configured to shift the lensassembly 252 to adjust the focus of the received light on the imagesensor 254.

At 1114, the apparatus may receive second image data of a subsequentframe of the scene, as described in connection with the examples inFIGS. 2 to 10 . For example, the processing unit 220 and/or the imageprocessor 600 may receive the second image data from the image sensor254. In some examples, the second image data may include focus data fromthe selected subset of the set of configurable pixels and imaging datafrom the remaining pixels of the plurality of pixels of the imagesensor.

FIG. 12 is a flowchart 1200 of a method implementing object detectionand/or recognition techniques, in accordance with one or more techniquesdisclosed herein. At 1202, the apparatus may apply object detectionand/or recognition techniques, as described in connection with theexamples in FIGS. 2 to 10 . For example, the object handling component610 may be configured to apply the object detection and/or recognitiontechniques. In some examples, the object detection and/or recognitiontechniques may be configured to detect and/or recognize differentclasses of objects. For example, the object detection and/or recognitiontechniques may be configured to detect and/or recognize an object ofinterest (e.g., a target item) and/or an object of non-interest. In someexamples, the object detection and/or recognition techniques may beconfigured to sub-classify detected and/or recognized objects. Forexample, the object detection and/or recognition techniques may beconfigured to detect and/or recognize an object of high interest (e.g.,a target item, such as a face) and/or may be configured to detect and/orrecognize an object of low interest (e.g., a microphone).

At 1204, the apparatus may determine whether an identified region is aregion of interest or a region of non-interest based on the detectedand/or recognized object, as described in connection with the examplesin FIGS. 2 to 10 . For example, the object handling component 610 may beconfigured to determine whether the identified region is a region ofinterest or a region of non-interest based on the object detected and/orrecognized within the identified region. In some examples, theidentified regions may be associated with different levels of interestor non-interest based on the corresponding class (or sub-class) of theobject detected and/or recognized within the identified region.

If, at 1204, the apparatus determines that the identified region is aregion of non-interest (e.g., the object detected and/or recognizedwithin the identified region is of non-interest), then, at 1206, theapparatus may determine to decrease the density of focus pixelsassociated with the identified region, as described in connection withthe examples in FIGS. 2 to 10 . For example, the object handlingcomponent 610 may be configured to determine to decrease the density offocus pixels within the image sensor 254 that are associated with theidentified region by changing the configuration of one or moreconfigurable pixels to operate as imaging pixels. Control then proceedsto 1210 to determine whether there is another identified region anddetected and/or recognized object to process.

If, at 1204, the apparatus determines that the identified region is aregion of interest (e.g., the detected and/or recognized object withinthe identified region is of interest), then, at 1208, the apparatus maydetermine to increase the density of focus pixels associated with theidentified region, as described in connection with the examples in FIGS.2 to 10 . For example, the object handling component 610 may beconfigured to determine to increase the density of focus pixels withinthe image sensor 254 that are associated with the identified region. Insome examples, the apparatus may determine to apply different densitylevels of focus pixels to different regions of interest based on, forexample, sub-classifications of the detected and/or recognized objects.For example, the apparatus may set a first density level of focus pixelsassociated with a region of interest including an object of highinterest (e.g., a face) and may set a second density level of focuspixels associated with a region of interest including an object of lowinterest (e.g., a microphone). In some such examples, the region ofinterest with the first density level may be relatively more dense offocus pixels than the region of interest with the second density level.

At 1210, the apparatus may determine whether there is another identifiedregion to process based on the detected and/or recognized object, asdescribed in connection with the examples in FIGS. 2 to 10 . Forexample, the object handling component 610 may be configured todetermine whether there is another identified region to process.

If, at 1210, the apparatus determines that there is another identifiedregion to process, control returns to 1204 to determine whether theidentified region is a region of interest or a region of non-interestbased on the object detected and/or recognized within the identifiedregion.

If, at 1210, the apparatus determines that there is not anotheridentified region to process, then, at 1212, the apparatus may apply thepixel configurations, as described in connection with the examples inFIGS. 2 to 10 . For example, the processing unit 220 and/or the imageprocessor 600 may be configured to apply the pixel configurations to theconfigurable pixels of the image sensor to increase or decrease thedensity of focus pixels associated with an identified region.

FIG. 13 is a flowchart 1300 of a method implementing focus dataorientation detection techniques, in accordance with one or moretechniques disclosed herein. At 1302, the apparatus may perform PDanalysis on focus data of a frame, as described in connection with theexamples in FIGS. 2 to 10 . For example, the orientation handlingcomponent 620 may be configured to perform the PD analysis on the focusdata of the frame. In some examples, the apparatus may output confidencelevels for respective regions of interest of the frame.

At 1304, the apparatus may determine whether an output confidence levelfor a region of interest satisfies a confidence threshold, as describedin connection with the examples in FIGS. 2 to 10 . For example, theorientation handling component 620 may be configured to determinewhether the output confidence level for the region of interest isgreater than or equal to the confidence threshold.

If, at 1304, the apparatus determines that the output confidence levelfor the region of interest does not satisfy the confidence threshold(e.g., is less than the confidence threshold), then, at 1306, theapparatus may determine to change a focus orientation for the focuspixels associated with the region of interest. For example, theorientation handling component 620 may be configured to change the focusorientation for the focus pixels associated with the region of interestfrom a first focus orientation to a second focus orientation. It may beappreciated that in some examples, the orientation handling component620 may be configured to determine to change the focus orientation for asubset of the focus pixels associated with the region of interest.Control then proceeds to 1310 to determine whether there is anotherconfidence level and region of interest to process.

If, at 1304, the apparatus determines that the output confidence levelfor the region of interest satisfies the confidence threshold (e.g., isgreater than or equal to the confidence threshold), then, at 1308, theapparatus may determine to maintain the focus orientation for the focuspixels associated with the region of interest. For example, theorientation handling component 620 may be configured to maintain thecurrent focus orientation for the focus pixels associated with theregion of interest.

At 1310, the apparatus may determine whether there is another confidencelevel and region of interest to process, as described in connection withthe examples in FIGS. 2 to 10 . For example, the orientation handlingcomponent 620 may be configured to determine whether there is anotherconfidence level and region of interest to process. If, at 1310, theapparatus determines that there is another confidence level and regionof interest to process, control returns to 1304 to determine whether theconfidence level satisfies the confidence threshold.

If, at 1310, the apparatus determines that there is not anotherconfidence level and region of interest to process, then, at 1312, theapparatus may apply the pixel configurations, as described in connectionwith the examples in FIGS. 2 to 10 . For example, the processing unit220 and/or the image processor 600 may be configured to apply the pixelconfigurations to the configurable pixels of the image sensor to changethe focus orientation of one or more of the focus pixels.

FIG. 14 is a flowchart 1400 of a method implementing saturationdetection techniques, in accordance with one or more techniquesdisclosed herein. At 1402, the apparatus may apply saturated areadetection techniques to detect one or more saturated regions in a frame,as described in connection with the examples in FIGS. 2 to 10 . Forexample, the saturation handling component 630 may be configured toapply saturated area detection techniques to detect the one or moresaturated regions in the frame.

At 1404, the apparatus may determine whether a detected saturated areaoverlaps with a region of interest, as described in connection with theexamples in FIGS. 2 to 10 . For example, the saturation handlingcomponent 630 may be configured to determine whether a detectedsaturated area overlaps with a region of interest. If, at 1404, theapparatus determines that the detected saturated area does not overlapwith a region of interest, then control proceeds to 1408 to determinewhether there is another saturated area and region of interest toprocess.

If, at 1404, the apparatus determines that the detected saturated areadoes overlap with a region of interest, then, at 1406, the apparatus maydetermine to reduce the density of focus pixels associated with thesaturated area, as described in connection with the examples in FIGS. 2to 10 . For example, the saturation handling component 630 may beconfigured to determine to reduce the density of focus pixels associatedwith the saturated area. In some examples, the apparatus may set a firstdensity level of focus pixels associated with the saturated area of theregion of interest and may set a second density level of focus pixelsfor the non-saturated area of the region of interest. In some examples,the second density level is greater than the first density level. Insome examples, the first density level may be zero so that theconfigurable pixels associated with the saturated area may be configuredto operate as imaging pixels. Control then proceeds to 1408 to determinewhether there is another detected saturated area to process.

At 1408, the apparatus may determine whether there is another detectedsaturated area to process, as described in connection with the examplesin FIGS. 2 to 10 . For example, the saturation handling component 630may be configured to determine whether there is another detectedsaturated area to process. If, at 1408, the apparatus determines thatthere is another detected saturated edge to process, control returns to1404 to determine whether the detected saturated area overlaps with aregion of interest.

If, at 1408, the apparatus determines that there is not another detectedsaturated area to process, then, at 1410, the apparatus may apply thepixel configurations, as described in connection with the examples inFIGS. 2 to 10 . For example, the processing unit 220 and/or the imageprocessor 600 may be configured to apply the pixel configurations to theconfigurable pixels of the image sensor.

FIG. 15 is a flowchart 1500 of a method implementing edge detectiontechniques, in accordance with one or more techniques disclosed herein.In the illustrated example, the apparatus may apply the edge detectiontechniques when the scene corresponds to a low-light environment andwhere applying binning techniques may be beneficial in improving thequality of the image.

At 1502, the apparatus may apply edge detection techniques to detect oneor more edges in a frame, as described in connection with the examplesin FIGS. 2 to 10 . For example, the edge handling component 640 may beconfigured to apply edge detection techniques. As described above, theedge detection techniques may be configured to detect a change inluminance or a change in color between proximate pixels. In someexamples, the apparatus may apply a threshold change in luminance orcolor to determine whether a change in luminance or a change in coloroccurred. In some examples, the edge may represent the boundary betweentwo or more objects (e.g., a desk and a wall) or a boundary betweendifferent areas of an object (e.g., a screen of a display monitor and achassis of the display monitor).

At 1504, the apparatus may determine whether a detected edge overlapswith a region of interest, as described in connection with the examplesin FIGS. 2 to 10 . For example, the edge handling component 640 may beconfigured to determine whether a detected edge overlaps with a regionof interest. If, at 1504, the apparatus determines that the detectededge does not overlap with a region of interest, then control proceedsto 1510 to determine whether there is another edge and region ofinterest to process.

If, at 1504, the apparatus determines that the detected edge doesoverlap with a region of interest, then, at 1506, the apparatus maydetermine to increase a quantity of focus pixels associated with a firstsub-region of interest based on the edge, as described in connectionwith the examples in FIGS. 2 to 10 . For example, the edge handlingcomponent 640 may be configured to determine to increase the quantity offocus pixels associated with the first sub-region of interest. In someexamples, the first focus pixels associated with the first sub-region ofinterest may correspond to a same object (e.g., the screen of a displaymonitor) and/or have a similar luminance or color.

At 1508, the apparatus may determine to increase a quantity of focuspixels associated with a second sub-region of interest based on theedge, as described in connection with the examples in FIGS. 2 to 10 .For example, the edge handling component 640 may be configured todetermine to increase the quantity of focus pixels associated with thesecond sub-region of interest. In some examples, the second focus pixelsassociated with the second sub-region of interest may correspond to asame object (e.g., the chassis of a display monitor) and/or have asimilar luminance or color.

At 1510, the apparatus may determine whether there is another detectededge and region of interest to process, as described in connection withthe examples in FIGS. 2 to 10 . For example, the edge handling component640 may be configured to determine whether there is another detectededge and region of interest to process. If, at 1510, the apparatusdetermines that there is another detected edge and region of interest toprocess, control returns to 1504 to determine whether the detected edgeoverlaps with a region of interest.

If, at 1510, the apparatus determines that there is not another detectededge and region of interest to process, then, at 1512, the apparatus mayapply the pixel configurations, as described in connection with theexamples in FIGS. 2 to 10 . For example, the processing unit 220 and/orthe image processor 600 may be configured to apply the pixelconfigurations to the configurable pixels of the image sensor.

The subject matter described herein can be implemented to realize one ormore benefits or advantages. For instance, the described imageprocessing techniques can be used by an application processor (e.g., anISP, a CPU, a GPU, a display processor, a DPU, a video processor, orsome other processor that can perform image processing) to implement theemploying of a PDAF optical system including pixels that may beconfigured to operate as focus pixels or imaging pixels to improve PDAFprocessing, reduce the likelihood of less beneficial information forPDAF purposes being transmitted, reduce the load of a processing unit(e.g., any processing unit configured to perform one or more techniquesdisclosed herein, such as an image processor), and/or reduce powerconsumption of the processing unit.

In accordance with this disclosure, the term “or” may be interrupted as“and/or” where context does not dictate otherwise. Additionally, whilephrases such as “one or more” or “at least one” or the like may havebeen used for some features disclosed herein but not others, thefeatures for which such language was not used may be interpreted to havesuch a meaning implied where context does not dictate otherwise.

In one or more examples, the functions described herein may beimplemented in hardware, software, firmware, or any combination thereof.For example, although the term “processing unit” has been usedthroughout this disclosure, such processing units may be implemented inhardware, software, firmware, or any combination thereof. If anyfunction, processing unit, technique described herein, or other moduleis implemented in software, the function, processing unit, techniquedescribed herein, or other module may be stored on or transmitted overas one or more instructions or code on a computer-readable medium.Computer-readable media may include computer data storage media orcommunication media including any medium that facilitates transfer of acomputer program from one place to another. In this manner,computer-readable media generally may correspond to (1) tangiblecomputer-readable storage media, which is non-transitory or (2) acommunication medium such as a signal or carrier wave. Data storagemedia may be any available media that can be accessed by one or morecomputers or one or more processors to retrieve instructions, codeand/or data structures for implementation of the techniques described inthis disclosure. By way of example, and not limitation, suchcomputer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or otheroptical disk storage, magnetic disk storage or other magnetic storagedevices. Disk and disc, as used herein, includes compact disc (CD),laser disc, optical disc, digital versatile disc (DVD), floppy disk andBlu-ray disc where disks usually reproduce data magnetically, whilediscs reproduce data optically with lasers. Combinations of the aboveshould also be included within the scope of computer-readable media. Acomputer program product may include a computer-readable medium.

The code may be executed by one or more processors, such as one or moredigital signal processors (DSPs), general purpose microprocessors,application specific integrated circuits (ASICs), arithmetic logic units(ALUs), field programmable logic arrays (FPGAs), or other equivalentintegrated or discrete logic circuitry. Accordingly, the term“processor,” as used herein may refer to any of the foregoing structureor any other structure suitable for implementation of the techniquesdescribed herein. Also, the techniques could be fully implemented in oneor more circuits or logic elements.

The techniques of this disclosure may be implemented in a wide varietyof devices or apparatuses, including a wireless handset, an integratedcircuit (IC) or a set of ICs, e.g., a chip set. Various components,modules or units are described in this disclosure to emphasizefunctional aspects of devices configured to perform the disclosedtechniques, but do not necessarily need realization by differenthardware units. Rather, as described above, various units may becombined in any hardware unit or provided by a collection ofinteroperative hardware units, including one or more processors asdescribed above, in conjunction with suitable software and/or firmware.

The following examples are illustrative only and may be combined withaspects of other embodiments or teachings described herein, withoutlimitation.

Example 1 is an apparatus comprising: an image sensor comprising aplurality of pixels, the plurality of pixels including a set of pixelsconfigurable to be imaging pixels or focus pixels, the image sensorconfigured to generate image data of a scene based on received light atthe plurality of pixels; and a processor coupled to the image sensor andconfigured to: receive first image data of a first frame of the scene;determine at least one region of interest or region of non-interest ofthe first frame; select, based on the determined at least one region ofinterest or region of non-interest, a subset of the set of pixels to befocus pixels; and cause the selected subset of the set of pixels tooperate as focus pixels.

In Example 2, the apparatus of Example 1 further includes that theprocessor is further configured to: receive, from the image sensor,second image data of a second frame, the second image data includingfocus data from the selected subset of the set of pixels and imagingdata from the remaining pixels of the plurality of pixels.

In Example 3, the apparatus of any of Example 1 or Example 2 furtherincludes a lens assembly configured to focus the received light on theimage sensor, wherein the processor is configured to shift the lensassembly to adjust the focus of the received light on the image sensorbased on the second image data.

In Example 4, the apparatus of any of Examples 1 to 3 further includesthat the processor is configured to select the subset of the set ofpixels to be focus pixels based on a determined region of interest by:increasing a density of focus pixels associated with the determinedregion of interest from a first density of focus pixels associated withthe determined region of interest when the first image data is receivedto a second density of focus pixels.

In Example 5, the apparatus of any of Examples 1 to 4 further includesthat the processor is configured to determine a density of focus pixelsassociated with the determined region of interest based on a target itemidentified in the determined region of interest.

In Example 6, the apparatus of any of Examples 1 to 5 further includesthat the determined region of interest or region of non-interestincludes a first region of interest and a second region of interest, andwherein the processor is configured to select a first density of focuspixels associated with the first region of interest and to select asecond density of focus pixels associated with the second region ofinterest, the second density being different than the first density.

In Example 7, the apparatus of any of Examples 1 to 6 further includesthat the processor is configured to select the subset of the set ofpixels to be focus pixels based on a determined region of non-interestby: decreasing a density of focus pixels associated with the determinedregion of non-interest from a first density of focus pixels associatedwith the determined region of non-interest when the first image data isreceived to a second density of focus pixels.

In Example 8, the apparatus of any of Examples 1 to 7 further includesthat the processor is configured to cause pixels in the set of pixelsassociated with the determined region of non-interest to operate asimaging pixels.

In Example 9, the apparatus of any of Examples 1 to 8 further includesthat the processor is configured to cause the selected subset of the setof pixels to operate as focus pixels by: setting a first focusorientation for each of the pixels of the selected subset of the set ofpixels based on at least one texture or edge detected within the firstframe.

In Example 10, the apparatus of any of Examples 1 to 9 further includesthat the processor is further configured to: determine whether aconfidence level of focus for second image data received from the imagesensor of a second frame satisfies a confidence threshold; and set asecond focus orientation for each of the pixels of the selected subsetof the set of pixels when the confidence level of focus does not satisfythe confidence threshold, the second focus orientation being differentthan the first focus orientation.

In Example 11, the apparatus of any of Examples 1 to 10 further includesthat the first focus orientation is one of an up-down orientation, aleft-right orientation, or a diagonal orientation.

In Example 12, the apparatus of any of Examples 1 to 11 further includesthat the determined region of interest or region of non-interestincludes a first region of interest and a second region of interest,wherein the first region of interest and the second region of interestare determined based on a level of light associated with each respectiveregion of interest, and wherein the first region of interest isassociated with a relatively lower level of light with respect to thesecond region of interest, and wherein the processor is furtherconfigured to: receive second image data of a second frame of the scene,the second image data including the same first region of interest andthe same second region of interest as the first frame; average data of afirst subset of focus pixels associated with the first region ofinterest; average data of a second subset of focus pixels associatedwith the second region of interest; and shift a lens assembly to adjustthe focus of the received light on the image sensor based on the averageof the data of the first subset of focus pixels and the average of thedata of the second subset of focus pixels.

In Example 13, the apparatus of any of Examples 1 to 12 further includesthat the processor is further configured to determine the first regionof interest and the second region of interest based on edge detectionassociated with the determined region of interest or region ofnon-interest.

In Example 14, the apparatus of any of Examples 1 to 13 further includesthat the processor is further configured to: increase a density of focuspixels associated with the first region of interest; and increase adensity of focus pixels associated with the second region of interest.

In Example 15, the apparatus of any of Examples 1 to 14 further includesthat the processor is further configured to: receive informationindicating the at least one region of interest or region of non-interestof the first frame, and wherein the processor is configured to determinethe at least one region of interest or region of non-interest based onthe received information.

In Example 16, the apparatus of any of Examples 1 to 15 further includesthat each pixel of the set of pixels includes: a photodiode; and anopacity transitioning material positioned above the photodiode, theopacity transitioning material including a first opacity transitioningmaterial portion arranged above a first section of the photodiode and asecond opacity transitioning material portion arranged above a secondsection of the photodiode, the first opacity transitioning materialportion and the second opacity transitioning material portion beingindependently configurable to be opaque or transparent.

In Example 17, the apparatus of any of Examples 1 to 16 further includesthat each pixel of the set of pixels is configured to operate as a focuspixel when one of the first opacity transitioning material portion orthe second opacity transitioning material portion is opaque and theother of the first opacity transitioning material portion or the secondopacity transitioning material portion is transparent, and the pixel isconfigured to operate as an imaging pixel when both of the first opacitytransitioning material portion and the second opacity transitioningmaterial portion are transparent.

In Example 18, the apparatus of any of Examples 1 to 17 further includesthat the opacity transitioning material further includes a third opacitytransitioning material portion arranged above a third section of thephotodiode and a fourth opacity transitioning material portion arrangedabove a fourth section of the photodiode, the first opacitytransitioning material portion, the second opacity transitioningmaterial portion, the third opacity transitioning material portion, andthe fourth opacity transitioning material portion being independentlyconfigurable to be opaque or transparent.

In Example 19, the apparatus of any of Examples 1 to 18 further includesthat the processor is further configured to set a focus orientation foreach of the focus pixels by configuring each of the first opacitytransitioning material portion, the second opacity transitioningmaterial portion, the third opacity transitioning material portion, andthe fourth opacity transitioning material portion to be independentlyopaque or transparent.

In Example 20, the apparatus of any of Examples 1 to 19 further includesthat the focus orientation is one of six different focus orientations,and wherein each of the six different focus orientations comprisessetting two of the opacity transitioning material portions to be opaqueand setting the remaining two opacity transitioning material portions tobe transparent.

Example 21 is a device including one or more processors and one or morememories in electronic communication with the one or more processorsstoring instructions executable by the one or more processors to cause asystem or an apparatus to implement a method as in any of Examples 1 to20.

Example 22 is a system or apparatus including means for implementing amethod or realizing an apparatus as in any of Examples 1 to 20.

Example 23 is a non-transitory computer-readable medium storinginstructions executable by one or more processors to cause the one ormore processors to implement a method as in any of Examples 1 to 20.

Various examples have been described. These and other examples arewithin the scope of the following claims.

What is claimed is:
 1. An apparatus, comprising: an image sensorcomprising a plurality of pixels, the plurality of pixels including aset of pixels each of which is configurable to switch between operationas an imaging pixel or as a focus pixel, the image sensor configured togenerate image data of a scene based on received light at the pluralityof pixels, at least one photodiode associated with each of the set ofpixels; and an opacity transitioning material positioned above the atleast one photodiode, the opacity transitioning material including afirst opacity transitioning material portion arranged above a firstsection of the at least one photodiode, a second opacity transitioningmaterial portion arranged above a second section of the at least onephotodiode, a third opacity transitioning material portion arrangedabove a third section of the at least one photodiode, and a fourthopacity transitioning material portion arranged above a fourth sectionof the at least one photodiode, the first opacity transitioning materialportion, the second opacity transitioning material portion, the thirdopacity transitioning material portion, and the fourth opacitytransitioning material portion each being independently configurable tobe opaque or transparent, wherein each pixel of the set of pixels isconfigured to operate as the focus pixel when one of the first opacitytransitioning material portion or the second opacity transitioningmaterial portion is opaque and another of the first opacitytransitioning material portion or the second opacity transitioningmaterial portion is transparent, and the pixel is configured to operateas the imaging pixel when both of the first opacity transitioningmaterial portion and the second opacity transitioning material portionare transparent.
 2. The apparatus of claim 1, further comprising: atleast one processor coupled to the image sensor and configured to: set afocus orientation for each of the focus pixels by configuring each ofthe first opacity transitioning material portion, the second opacitytransitioning material portion, the third opacity transitioning materialportion, and the fourth opacity transitioning material portion to beindependently opaque or transparent.
 3. The apparatus of claim 2,wherein the focus orientation is one of six different focusorientations, and wherein each of the six different focus orientationscomprises setting two of the opacity transitioning material portions tobe opaque and setting remaining two opacity transitioning materialportions to be transparent.
 4. The apparatus of claim 1, furthercomprising: at least one processor coupled to the image sensor andconfigured to: receive first image data of a first frame of the scene;determine at least one region of interest or region of non-interest ofthe first frame; select, based on the determined at least one region ofinterest or region of non-interest, a subset of the set of pixels tooperate as focus pixels; and cause the selected subset of the set ofpixels to operate as focus pixels.
 5. The apparatus of claim 4, whereinthe at least one processor is further configured to: receive, from theimage sensor, second image data of a second frame, the second image dataincluding focus data from the selected subset of the set of pixels andimaging data from remaining pixels of the plurality of pixels.
 6. Theapparatus of claim 5, further comprising: a lens assembly configured tofocus the received light on the image sensor, wherein the at least oneprocessor is configured to shift the lens assembly to adjust the focusof the received light on the image sensor based on the second imagedata.
 7. The apparatus of claim 4, wherein the at least one processor isconfigured to select the subset of the set of pixels to operate as focuspixels based on a determined region of interest by: increasing a densityof focus pixels associated with the determined region of interest from afirst density of focus pixels associated with the determined region ofinterest of the received first image data to a second density of focuspixels.
 8. The apparatus of claim 7, wherein the at least one processoris configured to determine the density of focus pixels associated withthe determined region of interest based on a target item identified inthe determined region of interest.
 9. The apparatus of claim 4, whereinthe determined region of interest or region of non-interest includes afirst region of interest and a second region of interest, and whereinthe at least one processor is configured to select a first density offocus pixels associated with the first region of interest and to selecta second density of focus pixels associated with the second region ofinterest, the second density being different than the first density. 10.The apparatus of claim 4, wherein the at least one processor isconfigured to select the subset of the set of pixels to operate as focuspixels based on a determined region of non-interest by: decreasing adensity of focus pixels associated with the determined region ofnon-interest from a first density of focus pixels associated with thedetermined region of non-interest of the received first image data to asecond density of focus pixels.
 11. The apparatus of claim 10, whereinthe at least one processor is configured to cause pixels in the set ofpixels associated with the determined region of non-interest to operateas imaging pixels.
 12. The apparatus of claim 4, wherein the at leastone processor is configured to cause the selected subset of the set ofpixels to operate as focus pixels by: setting a first focus orientationfor each of the pixels of the selected subset of the set of pixels basedon at least one texture or edge detected within the first frame.
 13. Theapparatus of claim 12, wherein the at least one processor is furtherconfigured to: determine whether a confidence level of focus for secondimage data received from the image sensor of a second frame satisfies aconfidence threshold; and set a second focus orientation for each of thepixels of the selected subset of the set of pixels when the confidencelevel of focus does not satisfy the confidence threshold, the secondfocus orientation being different than the first focus orientation. 14.The apparatus of claim 12, wherein the first focus orientation is one ofan up-down orientation, a left-right orientation, or a diagonalorientation.
 15. The apparatus of claim 4, wherein the determined regionof interest or region of non-interest includes a first region ofinterest and a second region of interest, wherein the first region ofinterest and the second region of interest are determined based on alevel of light associated with each respective region of interest, andwherein the first region of interest is associated with a relativelylower level of light with respect to the second region of interest, andwherein the at least one processor is further configured to: receivesecond image data of a second frame of the scene, the second image dataincluding a same first region of interest and a same second region ofinterest as the first frame; average data of a first subset of focuspixels associated with the first region of interest; average data of asecond subset of focus pixels associated with the second region ofinterest; and shift a lens assembly to adjust the focus of the receivedlight on the image sensor based on the average of the data of the firstsubset of focus pixels and the average of the data of the second subsetof focus pixels.
 16. The apparatus of claim 15, wherein the at least oneprocessor is further configured to determine the first region ofinterest and the second region of interest based on edge detectionassociated with the determined region of interest or region ofnon-interest.
 17. The apparatus of claim 15, wherein the at least oneprocessor is further configured to: increase first density of focuspixels associated with the first region of interest; and increase seconddensity of focus pixels associated with the second region of interest.18. The apparatus of claim 4, wherein the at least one processor isfurther configured to: receive information indicating the at least oneregion of interest or region of non-interest of the first frame, andwherein the at least one processor is configured to determine the atleast one region of interest or region of non-interest based on thereceived information.
 19. A method of operation at an image sensor,comprising: receiving light at a plurality of pixels of the imagesensor, the plurality of pixels including a set of pixels each of whichis configurable to switch between operation as an imaging pixel or as afocus pixel, the image sensor configured to generate image data of ascene based on the received light at the plurality of pixels; andconfiguring each pixel of the set of pixels as the focus pixel or as theimaging pixel, wherein the pixel is associated with at least onephotodiode, an opacity transitioning material is positioned above the atleast one photodiode, the opacity transitioning material including afirst opacity transitioning material portion arranged above a firstsection of the at least one photodiode, a second opacity transitioningmaterial portion arranged above a second section of the at least onephotodiode, a third opacity transitioning material portion arrangedabove a third section of the at least one photodiode, and a fourthopacity transitioning material portion arranged above a fourth sectionof the at least one photodiode, the first opacity transitioning materialportion, the second opacity transitioning material portion, the thirdopacity transitioning material portion, and the fourth opacitytransitioning material portion each being independently configurable tobe opaque or transparent, and the pixel is configured to operate as thefocus pixel when one of the first opacity transitioning material portionor the second opacity transitioning material portion is opaque andanother of the first opacity transitioning material portion or thesecond opacity transitioning material portion is transparent, and thepixel is configured to operate as the imaging pixel when both of thefirst opacity transitioning material portion and the second opacitytransitioning material portion are transparent.